Learning Reward Machines: A Study in Partially Observable Reinforcement Learning

Reinforcement Learning (RL) is a machine learning paradigm wherein an artificial agent interacts with environment the purpose of behaviour that maximizes expected cumulative reward it receives from environment. Reward machines (RMs) provide structured, automata-based representation function enables RL to decompose problem into structured subproblems can be efficiently learned via off-policy learning. Here we show RMs experience, instead being specified by user, and resulting decomposition used effectively solve partially observable problems. We pose task as discrete optimization where objective find RM decomposes set such combination their optimal memoryless policies policy for original problem. effectiveness this approach on three domains, significantly outperforms A3C, PPO, ACER, discuss its advantages, limitations, broader potential.1